The present invention generally relates to a learning machine for a data processing apparatus.
As the conventional learning machine is shown, for example, "A Parallel Neurocomputer Architecture toward Billion Connection Updates Per Second", International Joint Conference on Neural Network (January 1990).
FIG. 9 shows a block diagram of the conventional learning machine, which includes input and output signal registers 51, 52, 53 and 54, product-sum and sigmoid function computing units 55, 56 and 57, weight memories 58, 59 and 60 and ring length controlling units 61, 62 and 63. FIG. 10 shows a model of a learning machine to be realized with a structure shown in FIG. 9. In FIG. 10, reference numerals 64, 65, 66 and 67 are input terminals, reference numerals 68, 69, 70, 71 and 72 show multi-input single-output circuits, reference numeral 73 is an input layer, reference numeral 74 is a hidden layer, reference numeral 75 is an output layer. As shown in FIG. 10, a learning machine can be expressed in a model with a multi-input single-output circuits connected in hierachical structure. A layer composed of multi-input single-output circuits for outputting output signals from among multi-input single-output circuits connected in hierachical structure is called an output layer. Layers composed of multi-input single-output circuits excluding the output layer are called hidden layers. A layer composed of input terminals is called an input layer. Generally the hidden layer may be composed of multi-input single-output circuits constituting one layer, or may be composed of multi-input single-output circuits constituting a plurality of layers. FIG. 10 shows a case where the hidden layer is composed of multi-input single-output circuits constituting one layer. Generally the input terminals for constituting an input layer may be arbitrary in number. The multi-input single-output circuits for constituting the hidden layer and the output layer may be arbitrary in number. FIG. 10 shows a learning machine composed of a structure where the input layer 73 is composed of four input terminals, the hidden layer 74 is composed of three multi-input single-output circuits and the output layer 75 is composed of two multi-input single-output circuits. The multi-input single-output circuits 68, 69, 70, 71 and 72 of the respective layers output signals each having a saturation characteristic with respect to the sum of these products with individual weight being multiplied respectively by a plurality of input signals. Namely, an output signal Y.sub.j of a j-th multi-input single-output circuit is expressed by EQU Y.sub.j =fnc (.sub.i .SIGMA.(W.sub.ji X.sub.i)) (1)
Here X.sub.i is an output signal of an i-th multi-input single-output circuit in the proceeding layer. W.sub.ji is a weight to be multiplied when the output signal of an i-th of multi-input single-output circuit in the proceeding layer is inputted into the j-th multi-input single-output circuit. fnc ( ) is a sigmoid function having a saturation characteristic, and outputs for example ##EQU1## with respect to X.
In a block diagram of the conventional learning machine of FIG. 9, the weight memories 58, 59 and 60 store the weights to be multiplied in the multi-input single-output circuits 68, 69, 70, 71 in a model of FIG. 10. Input and output signal registers 51, 52, 53 and 54 store signals to be inputted from input terminals 64, 65, 66 and 67 or output signals of multi-input single-output circuits 68, 69, 70, 71 and 72. As a signal stored in the input and output signal register 54 is transferred to the input and output signal register 53 at the next machine-cycle and a signal stored in the input and output signal register 53 is transferred to the input and output signal register 52 at the next machine-cycle, signals stored in the input and output signal registers 51, 52, 53 and 54 are transferred in a sequence. A product-sum and sigmoid function computing units 55, 56 and 57 obtain a product-sum between weights stored in the weight memories 58, 59 and 60 and signals stored in the input and output signal registers 51, 52 and 53 so as to output a signal having a saturation characteristic with respect to product-sum in accordance with the (formula 2). The output signals of the product-sum and sigmoid function computing units 55, 56 and 57 are stored in the input and output signal registers 51, 52, 53 and 54. The ring length controlling units 61, 62 and 63 adjust the number of the input and output signal registers for transferring the stored signals in accordance with the number of the input signals and the number of the multi-input single-output circuits for constituting the hidden layer 74. Namely, in computing the product-sum of the multi-input single-output circuits of the hidden layer 74 the ring length is adjusted in the ring length controlling units 61, 62 and 63 so that the transfer operation of the signals among the input and output signal registers 51, 52, 53 and 54, as the number of the input signals is four. In computing the product-sum of the multi-input single-output circuits of the output layer 75, the ring length is adjusted in the ring length controlling units 61, 62 and 63 so that the signals are transferred among the input and output signal registers 51, 52 and 53, as the number of the multi-input single-output circuits of the hidden layer 74 is three.
FIG. 11 shows a block diagram of a product-sum and sigmoid function computing units 55, 56 and 57. In FIG. 11, reference numeral 76 is a multiplier, reference numeral 77 is a product-sum register, reference numeral 78 is an addition unit, reference numeral 79 is a sigmoid function computing element, reference numeral 80 is an input signal terminal and reference numeral 81 is an output signal terminal, reference numeral 82 is an input terminal of weights. The operations of the product-sum and sigmoid function computing units 55, 56 and 57 are shown hereinafter. A signal stored in the product-sum register 77 is initiated with zero. The multiplier 76 outputs to an addition unit 78 a product of a signal to be inputted from the input signal terminal 80 by a weight to be inputted from an input terminal of weights 82. The addition unit 78 obtains the sum of a product stored in the multiplication unit 76 outputs and a product-sum the product-sum register 77 so as to output it to a product-sum register 77. By repetition of an operation for obtaining such product and sum, a product-sum of signals to be inputted from the input signal terminal 80 and weights to be inputted from the input terminal of weights 82 is stored in the product-sum register 77. When the product-sum operation is completed, the sigmoid function computing element 79 outputs a signal having a saturation characteristic given in the (formula 2) with respect to a signal stored in the product-sum register 77. Therefore, a signal given in the (formula 2) is outputted from the output signal terminal 81.
FIG. 12 is a diagram for illustrating a parallel processing of the product-sum and sigmoid function computing units 55, 56 and 57 in obtaining the outputs of the multi-input single-output circuits 68, 69 and 70 of the hidden layer 74. In FIG. 12, X.sub.i (1.ltoreq.i.ltoreq.4) is an input signal, W.sub.ji (1.ltoreq.i.ltoreq.4, 1.ltoreq.j.ltoreq.3) is a weight to be multiplied by an input signal X.sub.i in a j-th multi-input single-output circuit of a hidden layer 74. In order to obtain the outputs of the multi-input single-output circuits 68, 69 and 70 of the hidden layer 74, the ring length is adjusted by the ring length controlling units 61, 62 and 63 so that the transfer of the signals may be effected among the input and output signal registers 51, 52, 53 and 54. At first, input signals X.sub.i (1.ltoreq.i.ltoreq.4) are loaded to the input and output signal registers 51, 52, 53 and 54. The product-sum registers of the product-sum and sigmoid function computing units 55, 56 and 57 are initiated with zero. The parallel processing of the product-sum and sigmoid function computing units 55, 56 and 57 at the next machine.cycle is shown in the (a) of FIG. 12. The product-sum and sigmoid function computing unit 55 obtains a product of a weight W.sub.11 stored in a weight memory 58 by an input signal X.sub.1 to be stored in an input and output signal register 51 so as to store it in its product-sum register. At the same time, the product-sum and sigmoid function computing unit 56 obtains of a product of W.sub.22 by X.sub.2, and the product-sum and sigmoid function computing unit 57 obtains a product of W.sub.33 by X.sub.3 so as to store them in the respective product-sum registers. The parallel processing of the product-sum and sigmoid function computing units 55, 56 and 57 at the next machine.cycle is shown in the (b) of FIG. 12. Signals stored in the input and output signal registers 51, 52, 53 and 54 are transferred in a sequence. The product-sum and sigmoid function computing unit 55 obtains a product W.sub.12 X.sub.2 of a signal X.sub.2 stored in the input and output signal register 51 by a weight W.sub.12 stored in a weight memory 58 so as to store in its product-sum register the sum with W.sub.11 X.sub.1 shown in the (formula 3). ##EQU2## At the same time, the product-sum and sigmoid function computing portions 56 and 57 respectively store in the product-sum register the product-sum shown in the (formula 4). ##EQU3## Likewise, in the following machine.cycles the signals stored in the input and output signal registers 51, 52, 53 and 54 are transferred in a sequence. The product-sum and sigmoid function computing units 55, 56 and 57 obtain product-sum of the weights stored in the weight memories 58, 59 and 60 by the signals stored in the input and output signal registers 51, 52, 53 and 54. Namely, the product-sum and sigmoid function computing units 55, 56 and 57 respectively obtain the product-sum in the first, second and third multi-input single-output circuits of the hidden layer. When the product-sum is obtained, the product-sum and sigmoid function computing units 55, 56 and 57 obtain signals each having a saturation characteristic given in the (formula 2) with respect to product-sum by a sigmoid function computing element so as to output them into the input and output signal registers 51, 52 and 53.
The output signals of the multi-input single-output circuits 68, 69 and 70 of the hidden layer 74 are obtained in a manner described hereinabove and are stored in the input and output signal registers 51, 52 and 53. In obtaining the outputs of the multi-input single-output circuits 71 and 72 of the output layer 75, the ring length is adjusted in the ring length controlling units 61, 62 and 63 so that the signals may be transferred among the input and output signal registers 51, 52 and 53 so as to keep the ring length consistent with the number of output signals (3 in this case) of the hidden layer. In the same way as obtaining of the output signals of the multi-input single-output circuits 68, 69 and 70 of the hidden layer 74, the outputs of the multi-input single-output circuits 71 and 72 of the output layer 75 are obtained by the parallel processing of the product-sum and sigmoid function computing units 55 and 56.
FIG. 13 shows a time chart showing the temporal sequence of the computing unit in the operation in the conventional learning machine. When the product-sum of the multi-input signal-output circuits 68, 69 and 70 of the hidden layer 74 are being obtained, the product-sum and sigmoid function computing units 55, 56 and 57 are in the operation, and the number of the product-sum and sigmoid function computing units operating at this time is in conformity of the number of the multi-input single-output circuits of the hidden layer. Time required to obtain the product-sum of the multi-input single-output circuit 68, 69 and 70 of the hidden layer 74 is EQU machine.cycle.times.number of signals in the input layer signals in the input layer (5)
Then, the sigmoid function of the multi-input single-output circuits of the hidden layer are obtained. The number of the product-sum and sigmoid function computing units operating at this time is equal to that of the multi-input single-output circuits of the hidden layer. When the product-sum of the multi-input single-output circuits 71 and 72 of the output layer 75 is obtained, the product-sum and sigmoid function computing units 55 and 56 are operating. The number of the product-sum and sigmoid function computing units operating at this time is in conformity with that of the multi-input single-output circuits of the output layer. Time required to obtain the product-sum of the multi-input single-output circuits 71 and 72 of the output layer 74 is EQU machine.cycle.times.number of signals in the hidden layer (6)
Then, the sigmoid function of the multi-input single-output circuits of the output layer is obtained. The number of the product-sum and sigmoid function computing units operating at this time is equal to that of the multi-input single-output circuits of the output layer. Time required for the output signal to be obtained from the input signal by the operation is EQU machine.cycle.times.(number of signals in the input layer+number of signals in the hidden layer)+duration of computing sigmoid function of the hidden layer+duration of computing sigmoid function of the output layer(7)
Then, the weights of the output layer are modified. The amount of weight modification of the output layer is abtained by the computation in the product-sum and sigmoid function computing units 55 and 56 so as to modify the weights. Further, a back-propagating signal .delta. of the hidden layer is obtained. Time required to obtain the weight modification of the output layer and the back-propagating signal .delta. of the hidden layer is EQU machine.cycle.times.number of signals in the hidden layer.times.3(8)
The weight modification of the hidden layer is effected by the operation in the product-sum and sigmoid function computing units 55, 56 and 57. The time required to do it is EQU machine.cycle.times.number of signals in the input layer.times.2(9)
Time required for the weight modification to be completed from a time point when the output signal of the output layer was obtained in the manner is EQU machine.cycle.times.(3.times.number of signals in the hidden layer+2.times.number of signals in the input layer) (10)
In the construction, the product-sum are abtained by the simultaneous operations of the product-sum computing elements number of which is equal to the number of the multi-input single-output circuits of one layer, so the product-sum computing elements are required by plurality equal to the number of the multi-input single-output circuits of the hidden layer or that of the output layer. Thus one of the problems of the PRIOR ART is that it results in larger scale of the circuits.
As the number of the product-sum and sigmoid function computing units for the parallel processing is less than that of the multi-input single-output circuits of the hidden layer or the output layer, the PRIOR ART has a problem that it cannot constitute such a learning machine with a number of multi-input single-output circuits in one layer exceed the number (3 in the conventional embodiment) of the previously prepared product-sum and sigmoid function computing units.
As the calculation of the amount of weight modification, the weight modification, the back-propagating signal .delta. of the hidden layer are sequentially carried out in a sequence by the product-sum and sigmoid function computing unit, the PRIOR ART has a problem that time required for the weight modification is long.